New RelicでAWSコストと使用状況レポート(CUR)をグラフ化してみた

sora

2023.09.25

この記事は公開されてから1年以上経過しています。情報が古い可能性がありますので、ご注意ください。

こんにちは、ゲームソリューション部のsoraです。
今回は、New RelicでAWSコストと使用状況レポート(CUR)をグラフ化してみたことについて書いていきます。

構成

AWSコストと使用状況レポート(AWS CUR)からS3にAWS請求レポートを発行します。
S3バケットへのレポート発行をトリガーとしてLambda関数が起動し、New Relicへレポートデータを送ります。
New Relic側では、Parsing Ruleによりレポートデータのmessageを変換して可視化します。

AWS CURのエクスポート

AWSマネージメントコンソールで請求ダッシュボード＞Cost & Usage reportsでレポートの作成をします。
基本的にそのままの設定で、圧縮タイプはGZIPとします。

New RelicのAPI Keyの作成

New Relic転送用のLambda関数で使用するNew RelicのAPIキーを作成します。
New Relicの自分の名前のところからAPI Keys＞Create a keyで、Key TypeをIngest - Licenseで作成します。
作成後はCopy keyでkey valueをコピーしておきます。

New Relic転送用のLambda関数の作成

New Relic転送用のLambda関数の作成します。
Lambda＞アプリケーションでアプリケーションを作成します。
サーバレスアプリケーションで[カスタムIAMロールまたはリソースポリシーを作成するアプリを表示する]にチェックを入れて検索し、NewRelic-log-ingestion-s3を選択します。
設定のNRLicenseKeyに先ほど作成したNew RelicのAPI Key valueを入力します。
次にS3へのオブジェクト配置をトリガーとするため、トリガー追加を行います。
バケットにはCURの出力先であるバケットを指定します。
これでAWS側での準備は完了です。

New RelicでのParsing Ruleの作成

ここからNew Relic側での設定を行います。
Logs＞ParsingでAWSから受け取ったレポートを変換するルールを作成します。
Filter logs based on NRQLにバケット名を入力します。（`aws.s3_bucket_name` = 'newrelic-log-test-20230919'）
※New Relic公式手順のバケット名を囲む記号が誤っているのか変換が上手くいかずに私はここで詰まりました。‘Amazon S3バケット名’ではなく'Amazon S3バケット名'で入力しました。私はParsing rule and outputには以下を入力します。

%{GREEDYDATA:log:csv({"columns": ["identity/LineItemId","identity/TimeInterval","bill/InvoiceId","bill/InvoicingEntity","bill/BillingEntity","bill/BillType","bill/PayerAccountId","bill/BillingPeriodStartDate","bill/BillingPeriodEndDate","lineItem/UsageAccountId","lineItem/LineItemType","lineItem/UsageStartDate","lineItem/UsageEndDate","lineItem/ProductCode","lineItem/UsageType","lineItem/Operation","lineItem/AvailabilityZone","lineItem/ResourceId","lineItem/UsageAmount","lineItem/NormalizationFactor","lineItem/NormalizedUsageAmount","lineItem/CurrencyCode","lineItem/UnblendedRate","lineItem/UnblendedCost","lineItem/BlendedRate","lineItem/BlendedCost","lineItem/LineItemDescription","lineItem/TaxType","lineItem/LegalEntity","product/ProductName","product/abdInstanceClass","product/alarmType","product/availability","product/availabilityZone","product/backupservice","product/bundle","product/bundleDescription","product/bundleGroup","product/callingType","product/capacitystatus","product/ciType","product/classicnetworkingsupport","product/clockSpeed","product/country","product/cputype","product/currentGeneration","product/dataTransferQuota","product/databaseEdition","product/databaseEngine","product/datatransferout","product/dedicatedEbsThroughput","product/deploymentOption","product/description","product/directorySize","product/directoryType","product/directoryTypeDescription","product/durability","product/ecu","product/endpointType","product/engine","product/engineCode","product/enhancedNetworkingSupported","product/equivalentondemandsku","product/eventType","product/feeCode","product/feeDescription","product/findingGroup","product/findingSource","product/findingStorage","product/freeTrial","product/freeUsageIncluded","product/fromLocation","product/fromLocationType","product/fromRegionCode","product/fusfCategory","product/gpuMemory","product/group","product/groupDescription","product/highAvailability","product/insightstype","product/instance","product/instanceFamily","product/instanceType","product/instanceTypeFamily","product/intelAvx2Available","product/intelAvxAvailable","product/intelTurboAvailable","product/invocation","product/license","product/licenseModel","product/lineType","product/location","product/locationType","product/lockeprofiles","product/logsDestination","product/marketoption","product/maxIopsBurstPerformance","product/maxIopsvolume","product/maxThroughputvolume","product/maxVolumeSize","product/maximumStorageVolume","product/memory","product/memorytype","product/messageDeliveryFrequency","product/messageDeliveryOrder","product/minVolumeSize","product/minimumStorageVolume","product/networkPerformance","product/normalizationSizeFactor","product/operatingSystem","product/operation","product/physicalProcessor","product/platopricingtype","product/platostoragetype","product/platousagetype","product/platovolumetype","product/preInstalledSw","product/primaryplaceofuse","product/processorArchitecture","product/processorFeatures","product/productFamily","product/protocol","product/queueType","product/region","product/regionCode","product/requestDescription","product/requestType","product/resourceType","product/rootvolume","product/routingTarget","product/routingType","product/runningMode","product/scanType","product/servicecode","product/servicename","product/sku","product/softwareIncluded","product/sourcetype","product/standardGroup","product/standardStorage","product/steps","product/storage","product/storageClass","product/storageMedia","product/storageType","product/tenancy","product/tenancySupport","product/tickettype","product/tiertype","product/timeWindow","product/toLocation","product/toLocationType","product/toRegionCode","product/transactionType","product/transferType","product/usageVolume","product/usagetype","product/uservolume","product/vcpu","product/version","product/volumeApiName","product/volumeType","product/vpcnetworkingsupport","pricing/RateCode","pricing/RateId","pricing/currency","pricing/publicOnDemandCost","pricing/publicOnDemandRate","pricing/term","pricing/unit","reservation/AmortizedUpfrontCostForUsage","reservation/AmortizedUpfrontFeeForBillingPeriod","reservation/EffectiveCost","reservation/EndTime","reservation/ModificationStatus","reservation/NormalizedUnitsPerReservation","reservation/NumberOfReservations","reservation/RecurringFeeForUsage","reservation/StartTime","reservation/SubscriptionId","reservation/TotalReservedNormalizedUnits","reservation/TotalReservedUnits","reservation/UnitsPerReservation","reservation/UnusedAmortizedUpfrontFeeForBillingPeriod","reservation/UnusedNormalizedUnitQuantity","reservation/UnusedQuantity","reservation/UnusedRecurringFee","reservation/UpfrontValue","savingsPlan/TotalCommitmentToDate","savingsPlan/SavingsPlanARN","savingsPlan/SavingsPlanRate","savingsPlan/UsedCommitment","savingsPlan/SavingsPlanEffectiveCost","savingsPlan/AmortizedUpfrontCommitmentForBillingPeriod","savingsPlan/RecurringCommitmentForBillingPeriod","resourceTags/user:Env","resourceTags/user:Name","resourceTags/user:STAGE","resourceTags/user:Use","resourceTags/user:company","resourceTags/user:team","resourceTags/user:user:Owner"], "noPrefix": false})}

しかし、エラーが出てParsing Ruleが作成できなかったため、New Relicに取り込まれたレポートからMessage部分を持ってきて、Choose your matching logのPaste logに貼り付けることで対処しました。
ログの検索は、LogsからNRQLを実行して取得しました。

SELECT
    `message`
FROM
    Log
WHERE
    allColumnSearch('S3バケット名', insensitive : true) SINCE 1 day ago
limit 10

New Relicでの取り込み+変換確認

New Relic側での設定も完了したため、レポートデータを確認していきます。
まずはSELECTでデータが取り込めているかの確認をします。

SELECT
    `message`
FROM
    Log
WHERE
    allColumnSearch('S3バケット名', insensitive : true) SINCE 1 day ago
limit 10

ログを選択して中身を見てみると、Parsing Ruleにて変換されていることがわかります。

New Relicでのグラフ化

変換されていることも確認できたため、グラフ化してみます。
New Relic公式ページ記載のものと若干選択する項目などを変えています。
明細項目の詳細は以下をご確認ください。
明細項目の詳細（AWS公式）

SELECT
    max(cost)
FROM
    (
        SELECT
            sum(numeric(`log.lineItem/UnblendedCost`)) as cost
        FROM
            Log
        WHERE
            `aws.s3_bucket_name` = 'S3バケット名' TIMESERIES 6 hour facet `log.lineItem/ProductCode`
    ) SINCE 7 days ago TIMESERIES 24 hours facet `log.lineItem/ProductCode`

stacked bar chart(積み上げ棒グラフ)にすると以下のようになります。